-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[UI Tests] - Add ensure_devices_found parameter to fastlane scan #14767
Open
jostnes
wants to merge
14
commits into
trunk
Choose a base branch
from
jostnes/testing-ui-tests-in-ci
base: trunk
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+6
−3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
📲 You can test the changes from this Pull Request in WooCommerce iOS by scanning the QR code below to install the corresponding build.
|
This reverts commit 17f66a6.
jostnes
changed the title
Testing in CI
[UI Tests] - Add ensure_devices_found parameter to fastlane scan
Dec 27, 2024
jostnes
added
type: task
An internally driven task.
category: ui tests
Related to UI testing.
labels
Dec 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The UI test suite is still flaky, with the test suite reliability over the last 7 days at only 62.5%. I checked the last few failing builds (merges to trunk) and found that:
WooCommerceUITests-Runner (2458) encountered an error (The test runner failed to initialize for UI testing. (Underlying Error: Timed out waiting for AX loaded notification))
WooCommerceUITests-Runner (2566) encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying Error: Crash: WooCommerceUITests-Runner (2566) <external symbol>))
The error
Underlying Error: Timed out waiting for AX loaded notification
looks to be related to the simulator not being ready before running the test, previously we added a check to make sure that the simulator is ready before the test starts. But missed the part that we're using fastlane's scan to run the test.Checking the fastlane doc, I found that there's a parameter that we could use to ensure that all simulators (since we are running with concurrency = 2) are ready before the test starts. This PR adds
ensure_devices_found: true
and replacedreset_simulator: true
(which can cause tests to start slower) toforce_quit_simulator: true
.I also updated the two timeout values of the two slowness screens from the default value of 20 seconds to 35 seconds. After running the test suite at least 5x, with the new parameters, I didn't encounter any failure with the error
Timed out waiting for AX loaded notification
. There are still timeouts happening to other screens due to CI slowness, but we can look at that again after the MacOS agent update and leave that out of the scope of this PR.Steps to reproduce
UI test suite is failing intermittently in CI.
Testing information
CI should be 🟢 . I also ran the test suite multiple times in CI and checked that this error didn't occur (testing builds):
Screenshots
RELEASE-NOTES.txt
if necessary.Reviewer (or Author, in the case of optional code reviews):
Please make sure these conditions are met before approving the PR, or request changes if the PR needs improvement: